scientific visualization Scientific visualization ( also spelled scientific visualisation) is an interdisciplinary branch of science concerned with the visualization of scientific phenomena.Michael Friendly (2008)"Milestones in the history of thematic cartography, stat ...

and

computer graphics Computer graphics deals with generating images with the aid of computers. Today, computer graphics is a core technology in digital photography, film, video games, cell phone and computer displays, and many specialized applications. A great de ...

, volume rendering is a set of techniques used to display a 2D projection of a 3D discretely

sampled Sample or samples may refer to: Base meaning * Sample (statistics), a subset of a population – complete data set * Sample (signal), a digital discrete sample of a continuous analog signal * Sample (material), a specimen or small quantity of so ...

data set A data set (or dataset) is a collection of data. In the case of tabular data, a data set corresponds to one or more database tables, where every column of a table represents a particular variable, and each row corresponds to a given record of the ...

, typically a 3D

scalar field In mathematics and physics, a scalar field is a function (mathematics), function associating a single number to every point (geometry), point in a space (mathematics), space – possibly physical space. The scalar may either be a pure Scalar ( ...

. A typical 3D data set is a group of 2D slice images acquired by a CT,

MRI Magnetic resonance imaging (MRI) is a medical imaging technique used in radiology to form pictures of the anatomy and the physiological processes of the body. MRI scanners use strong magnetic fields, magnetic field gradients, and radio waves ...

, or

MicroCT X-ray microtomography, like tomography and X-ray computed tomography, uses X-rays to create cross-sections of a physical object that can be used to recreate a virtual model ( 3D model) without destroying the original object. The prefix ''micro- ...

scanner. Usually these are acquired in a regular pattern (e.g., one slice for each millimeter of depth) and usually have a regular number of image

pixel In digital imaging, a pixel (abbreviated px), pel, or picture element is the smallest addressable element in a raster image, or the smallest point in an all points addressable display device. In most digital display devices, pixels are the smal ...

s in a regular pattern. This is an example of a regular volumetric grid, with each volume element, or

voxel In 3D computer graphics, a voxel represents a value on a regular grid in three-dimensional space. As with pixels in a 2D bitmap, voxels themselves do not typically have their position (i.e. coordinates) explicitly encoded with their values. Ins ...

represented by a single value that is obtained by sampling the immediate area surrounding the voxel. To render a 2D projection of the 3D data set, one first needs to define a

camera A camera is an Optics, optical instrument that can capture an image. Most cameras can capture 2D images, with some more advanced models being able to capture 3D images. At a basic level, most cameras consist of sealed boxes (the camera body), ...

in space relative to the volume. Also, one needs to define the opacity and color of every voxel. This is usually defined using an

RGBA RGBA stands for red green blue alpha. While it is sometimes described as a color space, it is actually a three-channel RGB color model supplemented with a fourth ''alpha channel''. Alpha indicates how opaque each pixel is and allows an image to ...

(for red, green, blue, alpha) transfer function that defines the RGBA value for every possible voxel value. For example, a volume may be viewed by extracting

isosurface An isosurface is a three-dimensional analog of an isoline. It is a surface that represents points of a constant value (e.g. pressure, temperature, velocity, density) within a volume of space; in other words, it is a level set of a continuous fu ...

s (surfaces of equal values) from the volume and rendering them as polygonal meshes or by rendering the volume directly as a block of data. The

marching cubes Marching cubes is a computer graphics algorithm, published in the 1987 SIGGRAPH proceedings by Lorensen and Cline, for extracting a polygonal mesh of an isosurface from a three-dimensional discrete scalar field (the elements of which are sometime ...

algorithm is a common technique for extracting an isosurface from volume data. Direct volume rendering is a computationally intensive task that may be performed in several ways. Another method of volume rendering is

Ray marching Ray marching is a class of rendering methods for 3D computer graphics where rays are traversed iteratively, effectively dividing each ray into smaller ray segments, sampling some function at each step. This function can encode volumetric data for ...

Scope

CT presentation as thin slice, projection and volume rendering

Volume rendering is distinguished from thin slice

tomography Tomography is imaging by sections or sectioning that uses any kind of penetrating wave. The method is used in radiology, archaeology, biology, atmospheric science, geophysics, oceanography, plasma physics, materials science, astrophysics, quantu ...

presentations, and is also generally distinguished from projections of 3D models, including

maximum intensity projection In scientific visualization, a maximum intensity projection (MIP) is a method for 3D data that projects in the visualization plane the voxels with maximum intensity that fall in the way of parallel rays traced from the viewpoint to the plane of ...

. Still, technically, all volume renderings become projections when viewed on a 2-dimensional display, making the distinction between projections and volume renderings a bit vague. Nevertheless, the epitomes of volume rendering models feature a mix of for example coloring and shading in order to create realistic and/or observable representations.

Direct volume rendering

A direct volume rendererMarc Levoy, "Display of Surfaces from Volume Data",

IEEE The Institute of Electrical and Electronics Engineers (IEEE) is a 501(c)(3) professional association for electronic engineering and electrical engineering (and associated disciplines) with its corporate office in New York City and its operation ...

CG&A, May 1988
Archive of Paper
/ref> requires every sample value to be mapped to opacity and a color. This is done with a "

transfer function In engineering, a transfer function (also known as system function or network function) of a system, sub-system, or component is a function (mathematics), mathematical function that mathematical model, theoretically models the system's output for ...

" which can be a simple ramp, a piecewise linear function or an arbitrary table. Once converted to an

RGBA color model RGBA stands for red green blue alpha. While it is sometimes described as a color space, it is actually a three-channel RGB color model supplemented with a fourth ''alpha channel''. Alpha indicates how opaque each pixel is and allows an image to ...

(for red, green, blue, alpha) value, the composed RGBA result is projected on the corresponding pixel of the frame buffer. The way this is done depends on the rendering technique. A combination of these techniques is possible. For instance, a shear warp implementation could use texturing hardware to draw the aligned slices in the

off-screen buffer A framebuffer (frame buffer, or sometimes framestore) is a portion of random-access memory (RAM) containing a bitmap that drives a video display. It is a memory buffer containing data representing all the pixels in a complete video frame. Modern ...

Volume ray casting

The technique of volume ray casting can be derived directly from the

rendering equation In computer graphics, the rendering equation is an integral equation in which the equilibrium radiance leaving a point is given as the sum of emitted plus reflected radiance under a geometric optics approximation. It was simultaneously introduced ...

. It provides results of very high quality, usually considered to provide the best image quality. Volume ray casting is classified as image based volume rendering technique, as the computation emanates from the output image, not the input volume data as is the case with object based techniques. In this technique, a ray is generated for each desired image pixel. Using a simple camera model, the ray starts at the center of projection of the camera (usually the eye point) and passes through the image pixel on the imaginary image plane floating in between the camera and the volume to be rendered. The ray is clipped by the boundaries of the volume in order to save time. Then the ray is sampled at regular or adaptive intervals throughout the volume. The data is interpolated at each sample point, the transfer function applied to form an RGBA sample, the sample is composited onto the accumulated RGBA of the ray, and the process repeated until the ray exits the volume. The RGBA color is converted to an RGB color and deposited in the corresponding image pixel. The process is repeated for every pixel on the screen to form the completed image.

Splatting

This is a technique which trades quality for speed. Here, every volume element is splatted, as Lee Westover said, like a snow ball, on to the viewing surface in back to front order. These splats are rendered as disks whose properties (color and transparency) vary diametrically in normal (

Gaussian Carl Friedrich Gauss (1777–1855) is the eponym of all of the topics listed below. There are over 100 topics all named after this German mathematician and scientist, all in the fields of mathematics, physics, and astronomy. The English eponymo ...

) manner. Flat disks and those with other kinds of property distribution are also used depending on the application.

Shear warp

The shear warp approach to volume rendering was developed by Cameron and Undrill, popularized by Philippe Lacroute and

Marc Levoy Marc Levoy is a computer graphics researcher and Professor Emeritus of Computer Science and Electrical Engineering at Stanford University, a vice president and Fellow at Adobe Inc., and (until 2020) a Distinguished Engineer at Google. He is noted ...

. In this technique, the

viewing transformation In computer graphics, a computer graphics pipeline, rendering pipeline or simply graphics pipeline, is a conceptual model that describes what steps a graphics system needs to perform to render a 3D scene to a 2D screen. Once ...

is transformed such that the nearest face of the volume becomes axis aligned with an off-screen image

data buffer In computer science, a data buffer (or just buffer) is a region of a memory used to temporarily store data while it is being moved from one place to another. Typically, the data is stored in a buffer as it is retrieved from an input device (such a ...

with a fixed scale of voxels to pixels. The volume is then rendered into this buffer using the far more favorable memory alignment and fixed scaling and blending factors. Once all slices of the volume have been rendered, the buffer is then warped into the desired orientation and scaled in the displayed image. This technique is relatively fast in software at the cost of less accurate sampling and potentially worse image quality compared to ray casting. There is memory overhead for storing multiple copies of the volume, for the ability to have near axis aligned volumes. This overhead can be mitigated using

run length encoding Run-length encoding (RLE) is a form of lossless data compression in which ''runs'' of data (sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original ...

Texture-based volume rendering

Many 3D graphics systems use

texture mapping Texture mapping is a method for mapping a texture on a computer-generated graphic. Texture here can be high frequency detail, surface texture, or color. History The original technique was pioneered by Edwin Catmull in 1974. Texture mapping ...

to apply images, or textures, to geometric objects. Commodity PC

graphics cards A graphics card (also called a video card, display card, graphics adapter, VGA card/VGA, video adapter, display adapter, or mistakenly GPU) is an expansion card which generates a feed of output images to a display device, such as a computer moni ...

are fast at texturing and can efficiently render slices of a 3D volume, with real time interaction capabilities.

Workstation A workstation is a special computer designed for technical or scientific applications. Intended primarily to be used by a single user, they are commonly connected to a local area network and run multi-user operating systems. The term ''workstat ...

GPU A graphics processing unit (GPU) is a specialized electronic circuit designed to manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device. GPUs are used in embedded systems, mobi ...

s are even faster, and are the basis for much of the production volume visualization used in

medical imaging Medical imaging is the technique and process of imaging the interior of a body for clinical analysis and medical intervention, as well as visual representation of the function of some organs or tissues (physiology). Medical imaging seeks to rev ...

, oil and gas, and other markets (2007). In earlier years, dedicated 3D texture mapping systems were used on graphics systems such as

Silicon Graphics Silicon Graphics, Inc. (stylized as SiliconGraphics before 1999, later rebranded SGI, historically known as Silicon Graphics Computer Systems or SGCS) was an American high-performance computing manufacturer, producing computer hardware and soft ...

InfiniteReality InfiniteReality refers to a 3D graphics hardware architecture and a family of graphics systems that implemented the aforementioned hardware architecture that was developed and manufactured by Silicon Graphics from 1996 to 2005. The InfiniteRealit ...

, HP Visualize FX

graphics accelerator A graphics processing unit (GPU) is a specialized electronic circuit designed to manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device. GPUs are used in embedded systems, mob ...

, and others. This technique was first described by Bill Hibbard and Dave Santek.Hibbard W., Santek D.
"Interactivity is the key"
''Chapel Hill Workshop on Volume Visualization'', University of North Carolina, Chapel Hill, 1989, pp. 39–43. These slices can either be aligned with the volume and rendered at an angle to the viewer, or aligned with the viewing plane and sampled from unaligned slices through the volume. Graphics hardware support for 3D textures is needed for the second technique. Volume aligned texturing produces images of reasonable quality, though there is often a noticeable transition when the volume is rotated.

Hardware-accelerated volume rendering

Due to the extremely parallel nature of direct volume rendering, special purpose volume rendering hardware was a rich research topic before

volume rendering became fast enough. The most widely cited technology was the VolumePro real-time ray-casting system, developed by

Hanspeter Pfister Hanspeter Pfister is a Swiss computer scientist. He is the An Wang Professor of Computer Science at the Harvard John A. Paulson School of Engineering and Applied Sciences and an affiliate faculty member of the Center for Brain Science at Harvard ...

and scientists at

Mitsubishi Electric Research Laboratories Mitsubishi Electric Research Laboratories (MERL) is a subsidiary of Mitsubishi Electric US Holdings, Inc., which, in its turn, is the principal subsidiary of Mitsubishi Electric in the United States. MERL is the North American arm of the Corpora ...

, which used high memory bandwidth and brute force to render using the ray casting algorithm. The technology was transferred to TeraRecon, Inc. and two generations of ASICs were produced and sold. The VP1000 was released in 2002 and the VP2000 in 2007. A recently exploited technique to accelerate traditional volume rendering algorithms such as ray-casting is the use of modern graphics cards. Starting with the programmable

pixel shader In computer graphics, a shader is a computer program that calculates the appropriate levels of light, darkness, and color during the rendering of a 3D scene - a process known as ''shading''. Shaders have evolved to perform a variety of spec ...

s, people recognized the power of parallel operations on multiple pixels and began to perform ''general-purpose computing on (the) graphics processing units'' (GPGPU). The

s are able to read and write randomly from video memory and perform some basic mathematical and logical calculations. These

SIMD Single instruction, multiple data (SIMD) is a type of parallel processing in Flynn's taxonomy. SIMD can be internal (part of the hardware design) and it can be directly accessible through an instruction set architecture (ISA), but it should ...

processors were used to perform general calculations such as rendering polygons and signal processing. In recent

generations, the pixel shaders now are able to function as

MIMD In computing, multiple instruction, multiple data (MIMD) is a technique employed to achieve parallelism. Machines using MIMD have a number of processors that function asynchronously and independently. At any time, different processors may be exe ...

processors (now able to independently branch) utilizing up to 1 GB of texture memory with floating point formats. With such power, virtually any algorithm with steps that can be performed in parallel, such as

volume ray casting Volume ray casting, sometimes called volumetric ray casting, volumetric ray tracing, or volume ray marching, is an image-based volume rendering technique. It computes 2D images from 3D volumetric data sets (3D scalar fields). Volume ray casting, ...

tomographic reconstruction Tomographic reconstruction is a type of multidimensional inverse problem where the challenge is to yield an estimate of a specific system from a finite number of projections. The mathematical basis for tomographic imaging was laid down by Johann ...

, can be performed with tremendous acceleration. The programmable

pixel shaders In computer graphics, a shader is a computer program that calculates the appropriate levels of light, darkness, and color during the rendering of a 3D scene - a process known as ''shading''. Shaders have evolved to perform a variety of speci ...

can be used to simulate variations in the characteristics of lighting, shadow, reflection, emissive color and so forth. Such simulations can be written using high level

shading language A shading language is a graphics programming language adapted to programming shader effects (characterizing surfaces, volumes, and objects). Such language forms usually consist of special data types, like "vector", "matrix", "color" and " normal". ...

Optimization techniques

The primary goal of optimization is to skip as much of the volume as possible. A typical medical data set can be 1 GB in size. To render that at 30 frame/s requires an extremely fast memory bus. Skipping voxels means that less information needs to be processed.

Empty space skipping

Often, a volume rendering system will have a system for identifying regions of the volume containing no visible material. This information can be used to avoid rendering these transparent regions.Sherbondy A., Houston M., Napel S.: ''Fast volume segmentation with simultaneous visualization using programmable graphics hardware.'' In Proceedings of

Visualization (2003), pp. 171–176.

Early ray termination

This is a technique used when the volume is rendered in front to back order. For a ray through a pixel, once sufficient dense material has been encountered, further samples will make no significant contribution to the pixel and so may be neglected.

Octree and BSP space subdivision

The use of hierarchical structures such as

octree An octree is a tree data structure in which each internal node has exactly eight children. Octrees are most often used to partition a three-dimensional space by recursively subdividing it into eight octants. Octrees are the three-dimensional ana ...

and BSP-tree could be very helpful for both compression of volume data and speed optimization of volumetric ray casting process.

Volume segmentation

CT angiography of the head without and with bone removal

Image segmentation In digital image processing and computer vision, image segmentation is the process of partitioning a digital image into multiple image segments, also known as image regions or image objects ( sets of pixels). The goal of segmentation is to simpl ...

is a manual or automatic procedure that can be used to section out large portions of the volume that one considers uninteresting before rendering, the amount of calculations that have to be made by ray casting or texture blending can be significantly reduced. This reduction can be as much as from O(n) to O(log n) for n sequentially indexed voxels. Volume segmentation also has significant performance benefits for other ray tracing algorithms. Volume segmentation can subsequently be used to highlight structures of interest.

Multiple and adaptive resolution representation

By representing less interesting regions of the volume in a coarser resolution, the data input overhead can be reduced. On closer observation, the data in these regions can be populated either by reading from memory or disk, or by

interpolation In the mathematical field of numerical analysis, interpolation is a type of estimation, a method of constructing (finding) new data points based on the range of a discrete set of known data points. In engineering and science, one often has a n ...

. The coarser resolution volume is resampled to a smaller size in the same way as a 2D mipmap image is created from the original. These smaller volume are also used by themselves while rotating the volume to a new orientation.

Pre-integrated volume rendering

Pre-integrated volume renderingMax N., Hanrahan P., Crawfis R.:
Area and volume coherence for efficient visualization of 3D scalar functions
'' In Computer Graphics (San Diego Workshop on Volume Visualization, 1990) vol. 24, pp. 27–33. is a method that can reduce sampling artifacts by pre-computing much of the required data. It is especially useful in hardware-accelerated applicationsLum E., Wilson B., Ma K.:
High-Quality Lighting and Efficient Pre-Integration for Volume Rendering
'' In Eurographics/

Symposium on Visualization 2004. because it improves quality without a large performance impact. Unlike most other optimizations, this does not skip voxels. Rather it reduces the number of samples needed to accurately display a region of voxels. The idea is to render the intervals between the samples instead of the samples themselves. This technique captures rapidly changing material, for example the transition from muscle to bone with much less computation.

Image-based meshing

Image-based meshing Image-based meshing is the automated process of creating computer models for computational fluid dynamics (CFD) and finite element analysis (FEA) from 3D image data (such as magnetic resonance imaging (MRI), computed tomography (CT) or microtomogra ...

is the automated process of creating computer models from 3D image data (such as

, CT, Industrial CT or microtomography) for computational analysis and design, e.g. CAD, CFD, and FEA.

Temporal reuse of voxels

For a complete display view, only one voxel per pixel (the front one) is required to be shown (although more can be used for smoothing the image), if animation is needed, the front voxels to be shown can be cached and their location relative to the camera can be recalculated as it moves. Where display voxels become too far apart to cover all the pixels, new front voxels can be found by ray casting or similar, and where two voxels are in one pixel, the front one can be kept.

List of related software

; Open source * 3D Slicer – a software package for scientific visualization and image analysis *

ClearVolume ClearVolume is an open source real-time live 3D visualization library designed for high-end volumetric light sheet microscopes. ClearVolume enables the live visualization of microscope data - allowing the biologists to immediately decide whether a ...

– a GPU ray-casting based, live 3D visualization library designed for high-end volumetric light sheet microscopes. *

ParaView ParaView is an open-source multiple-platform application for interactive, scientific visualization. It has a client–server architecture to facilitate remote visualization of datasets, and generates level of detail (LOD) models to maintain in ...

– a cross-platform, large data analysis and visualization application. ParaView users can quickly build visualizations to analyze their data using qualitative and quantitative techniques. ParaView is built on VTK (below). * Studierfenster (StudierFenster) – a free, non-commercial Open Science client/server-based Medical Imaging Processing (MIP) online framework. *

Vaa3D Vaa3D (in Chinese ‘挖三维’) is an Open Source visualization and analysis software suite created mainly by Hanchuan Peng and his team at Janelia Research Campus, HHMI and Allen Institute for Brain Science. The software performs 3D, 4D and 5D r ...

– a 3D, 4D and 5D volume rendering and image analysis platform for gigabytes and terabytes of large images (based on OpenGL) especially in the microscopy image field. Also cross-platform with Mac, Windows, and Linux versions. Include a comprehensive plugin interface and 100 plugins for image analysis. Also render multiple types of surface objects. *

VisIt Visit refer as go to see and spend time with socially. Visit may refer to: *State visit, a formal visit by a head of state to a foreign country *Conjugal visit, in which a prisoner is permitted to spend several hours or days in private with a visit ...

– a cross-platform interactive parallel visualization and graphical analysis tool for viewing scientific data. * Volume cartography – an open source software used in recovering the

En-Gedi Scroll The En-Gedi Scroll is an ancient Hebrew parchment found in 1970 at Ein Gedi, Israel. Radiocarbon testing dates the scroll to the third or fourth century CE (210–390 CE), although paleographical considerations suggest that the scrolls may date b ...

. * Voreen – a cross-platform rapid application development framework for the interactive visualization and analysis of multi-modal volumetric data sets. It provides GPU-based volume rendering and data analysis techniques *

VTK The Visualization Toolkit (VTK) is an open-source software system for 3D computer graphics, image processing and scientific visualization.''Visualization Handbook'', Academic Press, 2005, Chapter 30: the Visualization Toolkit/ref> VTK is distrib ...

– a general-purpose C++ toolkit for data processing, visualization, 3D interaction, computational geometry, with Python and Java bindings. Also, VTK.js provides a JavaScript implementation. ; Commercial *

Ambivu 3D Workstation AmbiVU 3D is a commercial medical imaging workstation that is compliant with the DICOM imaging and communications format. It is designed for high speed volume rendering of MRI, CT, PET, PET-CT and dual PET-CT datasets. AmbiVU 3D is available ...

– a medical imaging workstation that offers a range of volume rendering modes (based on OpenGL) * Amira – a 3D visualization and analysis software for scientists and researchers (in life sciences and biomedical) *

Imaris Bitplane is a provider of software for 3D and 4D image analysis for the life sciences. Founded in December 1992, Bitplane operates out of three offices in Zürich, Switzerland, Belfast, United Kingdom, and South Windsor, Connecticut, United Stat ...

– a scientific software module that delivers all the necessary functionality for data management, visualization, analysis, segmentation and interpretation of 3D and 4D microscopy datasets * MeVisLab – cross-platform software for medical image processing and visualization (based on OpenGL and Open Inventor) *

Open Inventor Open Inventor, originally IRIS Inventor, is a C++ object-oriented retained mode 3D graphics toolkit designed by SGI to provide a higher layer of programming for OpenGL. Its main goals are better programmer convenience and efficiency. Open Inven ...

– a high-level 3D API for 3D graphics software development (C++, .NET, Java) *

ScanIP Synopsys Simpleware ScanIP is a 3D image processing and model generation software program developed by Synopsys Inc. to visualise, analyse, quantify, segment and export 3D image data from magnetic resonance imaging (MRI), computed tomography ( ...

– an image processing and

image-based meshing Image-based meshing is the automated process of creating computer models for computational fluid dynamics (CFD) and finite element analysis (FEA) from 3D image data (such as magnetic resonance imaging (MRI), computed tomography (CT) or microtomogra ...

platform that can render scan data (MRI, CT, Micro-CT...) in 3D directly after import. *

tomviz tomviz is an open source software platform for reproducible volumetric visualization and data processing. The platform is designed for a wide range scientific applications but is especially tailored to high-resolution electron tomography, with feat ...

– a 3D visualization platform for scientists and researchers that can utilize Python scripts for advanced 3D data processing. * VoluMedic – a volume slicing and rendering software